Repository navigation
Kubernetes: manage the jump pod with a ReplicaSet - #4447
Merged
Merged
Conversation
The jump pod was a bare pod, reused for as long as it existed. Once evicted, for example due to node pressure, it stayed in the Failed phase and was never recreated, so every job in the project failed until someone deleted the pod by hand. The jump pod is now managed by a ReplicaSet, which replaces the pod after an eviction or a node removal. Not a Deployment: its ReplicaSet selects pods by the pod-template-hash label and thus cannot adopt the existing pod, and rollouts and rollbacks are not needed anyway. * run_job() creates the ReplicaSet if it doesn't exist. The ReplicaSet adopts a running legacy pod, a finished one is deleted. A pod created before #3891 has different labels and is left as is until it's gone. * A service created before #3891 selects pods by the old labels. Once the old pod is gone, the service is recreated with the new selector and the same NodePort, so that ssh_proxy of running jobs stays valid. * update_provisioning_data() lists jump pods by label and picks a ready one. It falls back to the name for a pod created before #3891 and fails right away if that pod is finished, as it's never replaced. * Creating the namespace, the ReplicaSet, and the service ignores 409 to tolerate concurrent run_job() calls. Notes: * The Role now needs get/create on replicasets and list on pods. Without the update, Kubernetes provisioning fails with 403. * Once the legacy pod is gone, servers before this commit recreate a bare pod and the service on each run_job(). The ReplicaSet adopts and deletes the pod, and the NodePort changes if proxy_jump.port is not set. Downgrading requires deleting the ReplicaSet. Part-of: #4407
un-def
added a commit
that referenced
this pull request
Nov 3, 2026
The jump pod requested no resources, so it was BestEffort, the first pod the kubelet evicts under node pressure. Since #4447, its ReplicaSet recreates it, but every eviction still drops all SSH sessions through the jump pod. * The jump pod container now requests and limits CPU and memory equally, which gives it the Guaranteed QoS class. The defaults, 500m and 1Gi, come from measuring the jump pod image: ~9 MiB plus ~1.6 MiB per tunneled SSH session, and 130-180 MiB/s of tunneled traffic at 500m. * The new proxy_jump.cpu and proxy_jump.memory properties override the defaults. Values must be Kubernetes quantities of at least 100m and 32Mi. * The container also requests 64Mi of ephemeral storage, without a limit. QoS doesn't cover disk pressure, under which the kubelet first evicts pods whose disk usage exceeds their ephemeral-storage request. * A jump pod that can't be scheduled, e.g., due to insufficient resources, is now logged as a warning with the scheduler's reason. Before, the only symptom was runs hitting the provisioning timeout. Notes: * New values don't apply to an existing jump pod. The docs describe how to apply them with kubectl. * A legacy jump pod that the ReplicaSet adopts on upgrade stays BestEffort until it's recreated. Fixes: #4407 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The jump pod was a bare pod, reused for as long as it existed. Once evicted, for example due to node pressure, it stayed in the Failed phase and was never recreated, so every job in the project failed until someone deleted the pod by hand.
The jump pod is now managed by a ReplicaSet, which replaces the pod after an eviction or a node removal. Not a Deployment: its ReplicaSet selects pods by the pod-template-hash label and thus cannot adopt the existing pod, and rollouts and rollbacks are not needed anyway.
Notes:
Part-of: #4407